{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "89cf2628",
   "metadata": {},
   "source": [
    "# State Estimation Example\n",
    "\n",
    "In this notebook we will walk through an example of state estimation calculation of `power-grid-model`.\n",
    "The following points are covered\n",
    " * Construction of the model\n",
    " * Run state estimation once, and its relevant function arguments\n",
    "\n",
    "\n",
    "It serves an example of how to use the Python API. For detailed API documentation, refer to \n",
    "[Python API reference](../api_reference/python-api-reference.md)\n",
    "and [Native Data Interface](../advanced_documentation/native-data-interface.md).\n",
    "\n",
    "\n",
    "## Example Network\n",
    "\n",
    "We use a simple network with 3 nodes, 1 source, 3 lines, and 2 loads. As shown below:\n",
    "\n",
    "```\n",
    " -----------------------line_8---------------\n",
    " |                                          |\n",
    "node_1 ---line_3--- node_2 ----line_5---- node_6\n",
    " |                    |                     |\n",
    "source_10          sym_load_4           sym_load_7\n",
    "```\n",
    "\n",
    "The 3 nodes are connected in a triangular way by 3 lines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ae11dc9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# some basic imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from power_grid_model import (\n",
    "    AngleMeasurementType,\n",
    "    AttributeType,\n",
    "    CalculationMethod,\n",
    "    CalculationType,\n",
    "    ComponentType,\n",
    "    DatasetType,\n",
    "    LoadGenType,\n",
    "    MeasuredTerminalType,\n",
    "    PowerGridModel,\n",
    "    initialize_array,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f983cef7",
   "metadata": {},
   "source": [
    "## Input Dataset\n",
    "\n",
    "We create input dataset by using the helper function `initialize_array`.\n",
    "Note the units of all input are standard SI unit without any prefix,\n",
    "i.e. the unit of voltage is volt (V), not kV.\n",
    "\n",
    "Please refer to [Components](../user_manual/components.md) for a detailed explanation of all component types and their input/output attributes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6f008736",
   "metadata": {},
   "outputs": [],
   "source": [
    "# node\n",
    "node = initialize_array(DatasetType.input, ComponentType.node, 3)\n",
    "node[AttributeType.id] = [1, 2, 6]\n",
    "node[AttributeType.u_rated] = [10.5e3, 10.5e3, 10.5e3]\n",
    "\n",
    "# line\n",
    "line = initialize_array(DatasetType.input, ComponentType.line, 3)\n",
    "line[AttributeType.id] = [3, 5, 8]\n",
    "line[AttributeType.from_node] = [1, 2, 1]\n",
    "line[AttributeType.to_node] = [2, 6, 6]\n",
    "line[AttributeType.from_status] = [1, 1, 1]\n",
    "line[AttributeType.to_status] = [1, 1, 1]\n",
    "line[AttributeType.r1] = [0.25, 0.25, 0.25]\n",
    "line[AttributeType.x1] = [0.2, 0.2, 0.2]\n",
    "line[AttributeType.c1] = [10e-6, 10e-6, 10e-6]\n",
    "line[AttributeType.tan1] = [0.0, 0.0, 0.0]\n",
    "line[AttributeType.i_n] = [1000, 1000, 1000]\n",
    "\n",
    "# load\n",
    "sym_load = initialize_array(DatasetType.input, ComponentType.sym_load, 2)\n",
    "sym_load[AttributeType.id] = [4, 7]\n",
    "sym_load[AttributeType.node] = [2, 6]\n",
    "sym_load[AttributeType.status] = [1, 1]\n",
    "sym_load[AttributeType.type] = [LoadGenType.const_power, LoadGenType.const_power]\n",
    "sym_load[AttributeType.p_specified] = [20e6, 10e6]\n",
    "sym_load[AttributeType.q_specified] = [5e6, 2e6]\n",
    "\n",
    "# source\n",
    "source = initialize_array(DatasetType.input, ComponentType.source, 1)\n",
    "source[AttributeType.id] = [10]\n",
    "source[AttributeType.node] = [1]\n",
    "source[AttributeType.status] = [1]\n",
    "source[AttributeType.u_ref] = [1.0]\n",
    "\n",
    "# voltage sensor\n",
    "sym_voltage_sensor = initialize_array(DatasetType.input, ComponentType.sym_voltage_sensor, 3)\n",
    "sym_voltage_sensor[AttributeType.id] = [11, 12, 13]\n",
    "sym_voltage_sensor[AttributeType.measured_object] = [1, 2, 6]\n",
    "sym_voltage_sensor[AttributeType.u_sigma] = [1.0, 1.0, 1.0]\n",
    "sym_voltage_sensor[AttributeType.u_measured] = [10489.37, 9997.32, 10102.01]\n",
    "sym_voltage_sensor[AttributeType.u_angle_measured] = [0.0, 0.0, 0.0]\n",
    "\n",
    "# power sensor\n",
    "sym_power_sensor = initialize_array(DatasetType.input, ComponentType.sym_power_sensor, 5)\n",
    "sym_power_sensor[AttributeType.id] = [14, 15, 16, 17, 18]\n",
    "sym_power_sensor[AttributeType.measured_object] = [3, 5, 8, 4, 6]\n",
    "sym_power_sensor[AttributeType.measured_terminal_type] = [\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.load,\n",
    "    MeasuredTerminalType.node,\n",
    "]\n",
    "sym_power_sensor[AttributeType.power_sigma] = [1.0e3, 1.0e3, 1.0e3, 1.0e3, 1.0e3]\n",
    "sym_power_sensor[AttributeType.p_measured] = [-1.66e07, -3.36e06, -1.33e07, 20e6, -10e6]\n",
    "sym_power_sensor[AttributeType.q_measured] = [-3.82e06, -1.17e06, -2.88e06, 5e6, -2e6]\n",
    "\n",
    "# current sensor\n",
    "sym_current_sensor = initialize_array(DatasetType.input, ComponentType.sym_current_sensor, 3)\n",
    "sym_current_sensor[AttributeType.id] = [19, 20, 21]\n",
    "sym_current_sensor[AttributeType.measured_object] = [3, 5, 8]\n",
    "sym_current_sensor[AttributeType.measured_terminal_type] = [\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "]\n",
    "sym_current_sensor[AttributeType.angle_measurement_type] = [\n",
    "    AngleMeasurementType.global_angle,\n",
    "    AngleMeasurementType.local_angle,\n",
    "    AngleMeasurementType.local_angle,\n",
    "]\n",
    "sym_current_sensor[AttributeType.i_sigma] = [100, 100, 100]\n",
    "sym_current_sensor[AttributeType.i_angle_sigma] = [0.1, 0.1, 0.1]\n",
    "sym_current_sensor[AttributeType.i_measured] = [978.814, 806.486, 776.753]\n",
    "sym_current_sensor[AttributeType.i_angle_measured] = [-0.20864859135578062, 1.368, 1.349]\n",
    "\n",
    "# all\n",
    "input_data = {\n",
    "    ComponentType.node: node,\n",
    "    ComponentType.line: line,\n",
    "    ComponentType.sym_load: sym_load,\n",
    "    ComponentType.source: source,\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor,\n",
    "    ComponentType.sym_current_sensor: sym_current_sensor,\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "157db2f0",
   "metadata": {},
   "source": [
    "## Validation (optional)\n",
    "For efficiency reasons, most of the data is not explicitly validated in the power grid model. However, in most cases, a state estimation will fail/crash if the data is invalid. Often with a low level error message that is hard to relate to the original objects. Therfore, it is recommended to always validate your data before constructing a PowerGridModel instance.\n",
    "\n",
    "The simplest and most effective way to validate your data is by using `assert_valid_input_data()` which will throw an error if it encounters any invalid data. See [Validation Examples](./Validation%20Examples.ipynb) for more detailed information on the validation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fab31f58",
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model.validation import assert_valid_input_data\n",
    "\n",
    "assert_valid_input_data(input_data=input_data, calculation_type=CalculationType.state_estimation)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "afae0224",
   "metadata": {},
   "source": [
    "## Construction\n",
    "\n",
    "The construction of the model is just calling the constructor of `PowerGridModel`. The default `system_frequency` is 50.0 Hz.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7ef134e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = PowerGridModel(input_data)\n",
    "# model = PowerGridModel(input_data, system_frequency=60.0)  # if you have another system frequency"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e3605c3e",
   "metadata": {},
   "source": [
    "## One-time State Estimation Calculation\n",
    "\n",
    "You can call the method `calculate_state_estimation` to do a one-time state estimation calculation based on the current network data in the model. In this case you should not specify the argument `update_data` as it is used for batch calculation.\n",
    "\n",
    "The following command executes a one-time state estimation calculation with (they are also the default values for those arguments)\n",
    "* Symmetric calculation\n",
    "* Iterative linear method\n",
    "* Error tolerance: 1e-8\n",
    "* Maximum number of iteration: 20."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "44c2de63",
   "metadata": {},
   "outputs": [],
   "source": [
    "output_data = model.calculate_state_estimation(\n",
    "    symmetric=True, error_tolerance=1e-8, max_iterations=20, calculation_method=CalculationMethod.iterative_linear\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d08aaf45",
   "metadata": {},
   "source": [
    "### Result Dataset\n",
    "\n",
    "We can also print one result dataset of node and line by converting the array to dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a581a36e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.998883</td>\n",
       "      <td>10488.272639</td>\n",
       "      <td>0.013952</td>\n",
       "      <td>3.120033e+07</td>\n",
       "      <td>6.755258e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952278</td>\n",
       "      <td>9998.918864</td>\n",
       "      <td>-0.009381</td>\n",
       "      <td>-2.002141e+07</td>\n",
       "      <td>-4.812473e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962198</td>\n",
       "      <td>10103.079560</td>\n",
       "      <td>-0.005095</td>\n",
       "      <td>-9.968972e+06</td>\n",
       "      <td>-1.955178e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.998883  10488.272639  0.013952  3.120033e+07  6.755258e+06\n",
       "1   2          1  0.952278   9998.918864 -0.009381 -2.002141e+07 -4.812473e+06\n",
       "2   6          1  0.962198  10103.079560 -0.005095 -9.968972e+06 -1.955178e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Line result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>loading</th>\n",
       "      <th>p_from</th>\n",
       "      <th>q_from</th>\n",
       "      <th>i_from</th>\n",
       "      <th>s_from</th>\n",
       "      <th>p_to</th>\n",
       "      <th>q_to</th>\n",
       "      <th>i_to</th>\n",
       "      <th>s_to</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.983966</td>\n",
       "      <td>1.736193e+07</td>\n",
       "      <td>3.929896e+06</td>\n",
       "      <td>979.903666</td>\n",
       "      <td>1.780115e+07</td>\n",
       "      <td>-1.663862e+07</td>\n",
       "      <td>-3.681085e+06</td>\n",
       "      <td>983.966281</td>\n",
       "      <td>1.704095e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.205962</td>\n",
       "      <td>-3.382792e+06</td>\n",
       "      <td>-1.131388e+06</td>\n",
       "      <td>205.961755</td>\n",
       "      <td>3.566977e+06</td>\n",
       "      <td>3.413781e+06</td>\n",
       "      <td>8.387981e+05</td>\n",
       "      <td>200.886407</td>\n",
       "      <td>3.515321e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>0.781260</td>\n",
       "      <td>1.383840e+07</td>\n",
       "      <td>2.825362e+06</td>\n",
       "      <td>777.480115</td>\n",
       "      <td>1.412388e+07</td>\n",
       "      <td>-1.338275e+07</td>\n",
       "      <td>-2.793976e+06</td>\n",
       "      <td>781.259502</td>\n",
       "      <td>1.367130e+07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized   loading        p_from        q_from      i_from  \\\n",
       "0   3          1  0.983966  1.736193e+07  3.929896e+06  979.903666   \n",
       "1   5          1  0.205962 -3.382792e+06 -1.131388e+06  205.961755   \n",
       "2   8          1  0.781260  1.383840e+07  2.825362e+06  777.480115   \n",
       "\n",
       "         s_from          p_to          q_to        i_to          s_to  \n",
       "0  1.780115e+07 -1.663862e+07 -3.681085e+06  983.966281  1.704095e+07  \n",
       "1  3.566977e+06  3.413781e+06  8.387981e+05  200.886407  3.515321e+06  \n",
       "2  1.412388e+07 -1.338275e+07 -2.793976e+06  781.259502  1.367130e+07  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sym_load result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "      <th>i</th>\n",
       "      <th>s</th>\n",
       "      <th>pf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2.002141e+07</td>\n",
       "      <td>4.812473e+06</td>\n",
       "      <td>1188.989174</td>\n",
       "      <td>2.059167e+07</td>\n",
       "      <td>0.972306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>9.968972e+06</td>\n",
       "      <td>1.955178e+06</td>\n",
       "      <td>580.539808</td>\n",
       "      <td>1.015889e+07</td>\n",
       "      <td>0.981305</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized             p             q            i             s  \\\n",
       "0   4          1  2.002141e+07  4.812473e+06  1188.989174  2.059167e+07   \n",
       "1   7          1  9.968972e+06  1.955178e+06   580.539808  1.015889e+07   \n",
       "\n",
       "         pf  \n",
       "0  0.972306  \n",
       "1  0.981305  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Node result\")\n",
    "display(pd.DataFrame(output_data[ComponentType.node]))\n",
    "\n",
    "print(\"Line result\")\n",
    "display(pd.DataFrame(output_data[ComponentType.line]))\n",
    "\n",
    "print(\"Sym_load result\")\n",
    "display(pd.DataFrame(output_data[ComponentType.sym_load]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2c7c914",
   "metadata": {},
   "source": [
    "### Calculation Method\n",
    "\n",
    "There are two calculation methods available. They are listed in the `CalculationMethod` enumeration as follows:\n",
    "* **Iterative linear method**: \n",
    "    * `CalculationMethod.iterative_linear`: All measurements are linearized before running the method.\n",
    "* **Newton Raphson method**: \n",
    "    * `CalculationMethod.newton_raphson`: traditional Newton-Raphson method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00be7ae5",
   "metadata": {},
   "source": [
    "**In the following, state estimation is performed with the Newton-Raphson method. Note the difference in the results for the nodes compared to the results from the iterative linear method.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "40e6d425",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999171</td>\n",
       "      <td>10491.294313</td>\n",
       "      <td>0.013903</td>\n",
       "      <td>3.114259e+07</td>\n",
       "      <td>6.983679e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952402</td>\n",
       "      <td>10000.219061</td>\n",
       "      <td>-0.009023</td>\n",
       "      <td>-1.997281e+07</td>\n",
       "      <td>-4.996913e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962376</td>\n",
       "      <td>10104.949737</td>\n",
       "      <td>-0.004879</td>\n",
       "      <td>-9.960845e+06</td>\n",
       "      <td>-2.000371e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.999171  10491.294313  0.013903  3.114259e+07  6.983679e+06\n",
       "1   2          1  0.952402  10000.219061 -0.009023 -1.997281e+07 -4.996913e+06\n",
       "2   6          1  0.962376  10104.949737 -0.004879 -9.960845e+06 -2.000371e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "output_data_NR = model.calculate_state_estimation(symmetric=True, calculation_method=CalculationMethod.newton_raphson)\n",
    "\n",
    "print(\"Node result\")\n",
    "display(pd.DataFrame(output_data_NR[ComponentType.node]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a90ab8fb",
   "metadata": {},
   "source": [
    "It is also possible to specify the standard deviations of active and reactive power measurements seperately. This is especially powerful when using the Newton Raphson method for state estimation. Refer to the [Documentation](../user_manual/calculations.md) for a detailed explanation. Below is an example of state estimation with seperate standard deviations for p and q measurements.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "699e31af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Node result\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.999144</td>\n",
       "      <td>10491.009558</td>\n",
       "      <td>0.013841</td>\n",
       "      <td>3.109941e+07</td>\n",
       "      <td>7.028384e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952418</td>\n",
       "      <td>10000.387584</td>\n",
       "      <td>-0.008985</td>\n",
       "      <td>-1.991963e+07</td>\n",
       "      <td>-5.002841e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962317</td>\n",
       "      <td>10104.325522</td>\n",
       "      <td>-0.004856</td>\n",
       "      <td>-9.973746e+06</td>\n",
       "      <td>-2.041416e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.999144  10491.009558  0.013841  3.109941e+07  7.028384e+06\n",
       "1   2          1  0.952418  10000.387584 -0.008985 -1.991963e+07 -5.002841e+06\n",
       "2   6          1  0.962317  10104.325522 -0.004856 -9.973746e+06 -2.041416e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# power sensor\n",
    "sym_power_sensor2 = initialize_array(DatasetType.input, ComponentType.sym_power_sensor, 5)\n",
    "sym_power_sensor2[AttributeType.id] = [14, 15, 16, 17, 18]\n",
    "sym_power_sensor2[AttributeType.measured_object] = [3, 5, 8, 4, 6]\n",
    "sym_power_sensor2[AttributeType.measured_terminal_type] = [\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.branch_from,\n",
    "    MeasuredTerminalType.branch_to,\n",
    "    MeasuredTerminalType.load,\n",
    "    MeasuredTerminalType.node,\n",
    "]\n",
    "sym_power_sensor2[AttributeType.p_measured] = [-1.66e07, -3.36e06, -1.33e07, 20e6, -10e6]\n",
    "sym_power_sensor2[AttributeType.q_measured] = [-3.82e06, -1.17e06, -2.88e06, 5e6, -2e6]\n",
    "# define different standard deviations for p and q measurements\n",
    "sym_power_sensor2[AttributeType.p_sigma] = [1.0e3, 1.0e4, 1.0e3, 1.0e4, 1.0e3]\n",
    "sym_power_sensor2[AttributeType.q_sigma] = [1.0e4, 1.0e3, 1.0e4, 1.0e3, 1.0e4]\n",
    "\n",
    "# all\n",
    "input_data2 = {\n",
    "    ComponentType.node: node,\n",
    "    ComponentType.line: line,\n",
    "    ComponentType.sym_load: sym_load,\n",
    "    ComponentType.source: source,\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor2,\n",
    "}\n",
    "\n",
    "model2 = PowerGridModel(input_data2)\n",
    "output_data_NR2 = model2.calculate_state_estimation(symmetric=True, calculation_method=CalculationMethod.newton_raphson)\n",
    "print(\"Node result\")\n",
    "display(pd.DataFrame(output_data_NR2[ComponentType.node]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e516b436",
   "metadata": {},
   "source": [
    "## Observability\n",
    "\n",
    "In order to perform a state estimation the number of measurements should be larger than or equal to the number of unknowns. For each node there are two unknowns: `AttributeType.u` and `AttributeType.u_angle`.\n",
    "\n",
    "$$n_{\\text{measurements}} >= n_{\\text{unknowns}}$$\n",
    "\n",
    "Where\n",
    "\n",
    "$$n_{\\text{unknowns}} = 2 n_{\\text{nodes}} - 1 $$\n",
    "\n",
    "And\n",
    "\n",
    "$$n_{\\text{measurements}} = n_{\\text{nodes\\_with\\_voltage\\_sensor\\_without\\_angle}} + 2 n_{\\text{nodes\\_with\\_voltage\\_sensor\\_with\\_angle}} + 2 n_{\\text{branches\\_with\\_power\\_sensor}} + 2 n_{\\text{nodes\\_without\\_any\\_appliances\\_connected}} + 2 n_{\\text{nodes\\_with\\_all\\_connected\\_appliances\\_measured\\_by\\_power\\_sensor}}$$\n",
    "\n",
    "In this formula, only measurements with non-negligible contributions should be considered. That means that measurements with infinite variances have vanishing contribution and are to be omitted, regardless of whether those divergences are due to infinite sigmas or unreasonably large ones. As a result, missing data can be representing by providing infinity as a standard deviation, and the underdetermined edge case is handled correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc598467",
   "metadata": {},
   "source": [
    "### Some missing measurements\n",
    "\n",
    "Few missing measurements may keep the data observable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b3e3fdc9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>energized</th>\n",
       "      <th>u_pu</th>\n",
       "      <th>u</th>\n",
       "      <th>u_angle</th>\n",
       "      <th>p</th>\n",
       "      <th>q</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.998939</td>\n",
       "      <td>10488.854697</td>\n",
       "      <td>0.011360</td>\n",
       "      <td>3.113644e+07</td>\n",
       "      <td>6.783581e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0.952378</td>\n",
       "      <td>9999.970147</td>\n",
       "      <td>-0.011969</td>\n",
       "      <td>-2.004965e+07</td>\n",
       "      <td>-4.759283e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>0.962303</td>\n",
       "      <td>10104.186383</td>\n",
       "      <td>-0.007496</td>\n",
       "      <td>-9.880385e+06</td>\n",
       "      <td>-2.039703e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  energized      u_pu             u   u_angle             p             q\n",
       "0   1          1  0.998939  10488.854697  0.011360  3.113644e+07  6.783581e+06\n",
       "1   2          1  0.952378   9999.970147 -0.011969 -2.004965e+07 -4.759283e+06\n",
       "2   6          1  0.962303  10104.186383 -0.007496 -9.880385e+06 -2.039703e+06"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sym_voltage_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_voltage_sensor, 1)\n",
    "# for each mutation, only one object is specified\n",
    "sym_voltage_sensor_update[AttributeType.id] = 13\n",
    "sym_voltage_sensor_update[AttributeType.u_sigma] = np.inf  # disable this sensor\n",
    "\n",
    "sym_power_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_power_sensor, 1)\n",
    "sym_power_sensor_update[AttributeType.id] = 18\n",
    "sym_power_sensor_update[AttributeType.power_sigma] = np.inf  # disable this sensor\n",
    "\n",
    "update_data = {\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor_update,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor_update,\n",
    "}\n",
    "\n",
    "# Permanent update\n",
    "model.update(update_data=update_data)\n",
    "output_data = model.calculate_state_estimation()\n",
    "display(pd.DataFrame(output_data[ComponentType.node]))\n",
    "\n",
    "# fully reinstigate the original model to prevent this cell from influencing re-runs.\n",
    "# A model update would be sufficient as well.\n",
    "model = PowerGridModel(input_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd22b716",
   "metadata": {},
   "source": [
    "### Unobservable case\n",
    "\n",
    "If too many measurements are invalid, the result may become unobservable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "445726f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not enough measurements available for state estimation.\n",
      "No voltage sensor found!\n",
      "\n",
      "\n",
      "Try validate_input_data() or validate_batch_data() to validate your data.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from power_grid_model.errors import PowerGridError\n",
    "\n",
    "sym_voltage_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_voltage_sensor, 3)\n",
    "sym_voltage_sensor_update[AttributeType.id] = sym_voltage_sensor[AttributeType.id]\n",
    "sym_voltage_sensor_update[AttributeType.u_sigma] = np.inf  # disable all sensors\n",
    "\n",
    "sym_power_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_power_sensor, 5)\n",
    "sym_power_sensor_update[AttributeType.id] = sym_power_sensor[AttributeType.id]\n",
    "sym_power_sensor_update[AttributeType.power_sigma] = np.inf  # disable all sensors\n",
    "\n",
    "sym_current_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_current_sensor, 3)\n",
    "sym_current_sensor_update[AttributeType.id] = sym_current_sensor[AttributeType.id]\n",
    "sym_current_sensor_update[AttributeType.i_sigma] = np.inf  # disable all sensors\n",
    "\n",
    "update_data = {\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor_update,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor_update,\n",
    "    ComponentType.sym_current_sensor: sym_current_sensor_update,\n",
    "}\n",
    "\n",
    "try:\n",
    "    # Permanent update\n",
    "    model.update(update_data=update_data)\n",
    "    output_data = model.calculate_state_estimation()\n",
    "    print(\"Success\")\n",
    "except PowerGridError as e:\n",
    "    print(e)\n",
    "finally:\n",
    "    # fully reinstigate the original model to prevent this cell from influencing re-runs.\n",
    "    # A model update would be sufficient as well.\n",
    "    model = PowerGridModel(input_data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3da1f2aa",
   "metadata": {},
   "source": [
    "## Batch calculation\n",
    "\n",
    "For state estimation a batch calculation can be performed in a similar manner as for powerflow calculations. Below is a short example provided for reference. For more a more detailed example about performing a batch calculation, please refer to the [Power Flow Example](Power%20Flow%20Example.ipynb) or visit the [Documentation](../user_manual/calculations.md)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c90cc3f8",
   "metadata": {},
   "source": [
    "### Create Update Dataset\n",
    "The following code creates an update dataset which runs state estimation for a batch with three different measurements for the symmetric voltage sensors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "303af910",
   "metadata": {},
   "outputs": [],
   "source": [
    "sym_voltage_sensor_update = initialize_array(\n",
    "    DatasetType.update, ComponentType.sym_voltage_sensor, (4, 3)\n",
    ")  # 4 scenarios, 3 objects per scenario\n",
    "# for each mutation, only one object is specified\n",
    "sym_voltage_sensor_update[AttributeType.id] = [[11, 12, 13]] * 4\n",
    "sym_voltage_sensor_update[AttributeType.u_measured] = [\n",
    "    [10402.31, 10023.68, 10095.40],\n",
    "    [10583.20, 9869.85, 10105.28],\n",
    "    [10482.87, 9850.85, 10140.77],\n",
    "    [10482.87, 9850.85, 10140.77],\n",
    "]\n",
    "sym_voltage_sensor_update[AttributeType.u_sigma][2, 2] = np.inf  # disable the third sensor of the third scenario\n",
    "\n",
    "sym_power_sensor_update = initialize_array(DatasetType.update, ComponentType.sym_power_sensor, (4, 1))\n",
    "sym_power_sensor_update[AttributeType.id] = [18]\n",
    "sym_power_sensor_update[AttributeType.power_sigma] = [\n",
    "    [1.0e3],\n",
    "    [1.0e3],\n",
    "    [1.0e3],\n",
    "    [np.inf],  # disables this sensor on the last scenario\n",
    "]\n",
    "\n",
    "update_data = {\n",
    "    ComponentType.sym_voltage_sensor: sym_voltage_sensor_update,\n",
    "    ComponentType.sym_power_sensor: sym_power_sensor_update,\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bbf1d832",
   "metadata": {},
   "source": [
    "### Validation (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "70f761a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from power_grid_model.validation import assert_valid_batch_data\n",
    "\n",
    "assert_valid_batch_data(\n",
    "    input_data=input_data, update_data=update_data, calculation_type=CalculationType.state_estimation\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "68d935f0",
   "metadata": {},
   "source": [
    "### Call Update Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4d9d5de2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[10465.44337926  9977.00924576 10080.55297677]\n",
      " [10480.47976245  9986.25423471 10093.14551907]\n",
      " [10415.61048312  9919.84715207 10025.79649187]\n",
      " [10451.69932534  9957.73806984 10066.52669313]]\n"
     ]
    }
   ],
   "source": [
    "batch_output_data = model.calculate_state_estimation(\n",
    "    update_data=update_data,\n",
    "    symmetric=True,\n",
    "    error_tolerance=1e-8,\n",
    "    max_iterations=20,\n",
    "    calculation_method=CalculationMethod.iterative_linear,\n",
    ")\n",
    "print(batch_output_data[ComponentType.node][AttributeType.u])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "power-grid-model",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}